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ABSTRACT 



A computer program was developed for solving equality 
and inequality constrained optimization problems by the 
Augmented Lagrange Multiplier method. The program was 
developed specifically for use in engineering design. 

The historical evolution and theoretical development of 
the multiplier method is presented. Several examples are 
used to demonstrate the effects of penalty parameters and 
multipliers on the convergence and accuracy of the method. 
Computational experience with variations to the method is 
documented. 

A brief literature search of the multiplier method's 
application to engineering design is summarized. The method 
is demonstrated with several mathematical and engineering 
examples. A comparison to classical penalty methods and the 
method of feasible directions was performed in each case. 
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I. 



INTRODUCTION 



Several methods currently exist which solve the function 
minimization problem. While each has its advantage in special- 
ized cases, no one method, including the multiplier method, is 
universally more efficient and accurate than the others. The 
Augmented Lagrange Multiplier method has been shown to be an 
efficient and accurate method of engineering design optimiza- 
tion. The multiplier method is particularly advantageous in 
nonlinear equality constrained problems. It has become 
increasingly popular in engineering design optimization 
because of its good convergence rate and its attractive 
theoretical properties. 

The general nonlinear problem is defined; 



Minimize f (X) 






(1.1a) 


subject to g^(X) < 


0 


• 

• 

• 

1 — 1 

II 

•H 


(1.1b) 


and hj (X) = 


0 


( j=l , . . . ,m<n) 


(1.1c) 


and ^ 


< 


xV (i=l, . . . ,n) 


(l.ld) 



where X is a vector of n design variables, f(X) is the ob- 
jective function, g(X) is a set of £ inequality constraint 
functions, and h(X) is a set of m equality constraint functions, 
and X^ and X^ are bounds on the design variables, referred to 
as side constraints. The problem is solved by creating a 
single augmented Lagrange function L(X,A), where A is a vector 
of n Lagrange multipliers. The optimal solution (X*,A*) is 
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found by alternately solving a series of unconstrained 
minimizations followed by a simple update of the multipliers, 

X. 

In this study, the unconstrained minimization is performed 
by the Davidon-Fletcher-Powell method [1] utilizing a combina- 
tion of the Golden Section and polynomial one-dimensional 
search procedures. The Davidon-Fletcher-Powell method provides 
good reliability and convergence properties. The Golden 
Section search provides a reliable means of reducing the 
search bounds. In most cases, polynomial interpolation 
obtains a more accurate minimum than Golden Section for 
sufficiently narrow bounds. The combination provided a 
reliable and accurate one-dimensional search. 

The multiplier method has significant advantages. As 
will be shown later, the inherent numerical ill-conditioning 
of more common penalty function methods is reduced in the 
multiplier method. Exact solutions, not possible in penalty 
methods, are also attainable. Finally, multiplier methods 
are not restricted to convex programming as are pure primal- 
dual methods. 

There have been numerous applications of the multiplier 
method in the various disciplines of engineering design 
optimization. Imai [2] effectively used the method in 
structural optimization. Fax and Mills [3], while limited 
to equality constraints, applied the method to heat exchanger 
optimization. Hedderich's [4] work in heat exchanger 
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optimization showed the need for a method which could treat 
equality constraints directly, in addition to inequality 
constraints. He showed the Constrained function Minimization 
Program, CONMIN [5] to be very inefficient for the equality 
constrained problem. It is his work that motivated this 
research of the multiplier method. 

It is the objective of this research to develop an opera- 
tional computer program applicable to the various disciplines 
of engineering design optimization. It is not the author's 
intent to develop a universally superior optimization program, 
but to develop a program to be incorporated into a library 
of optimization programs with flexibility in the choice of 
unconstrained minimization subprograms, one dimensional search 
subprograms, derivative evaluation subprograms, and conver- 
gence criterion. 

The theory and historical development of the multiplier 
method is presented, concluded by a multiplier method algorithm 
for nonlinear equality and inequality constrained problems. 
Experimentation with computational aspects is then summarized. 
Finally, a set of mathematical and engineering test cases 
are solved demonstrating the method's effectiveness, followed 
by conclusions and a discussion of results. 
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II. THE MULTIPLIER METHOD 



The development of the multiplier method and a brief 
mathematical background is presented in this chapter. 
Mathematical proofs are brief since computational aspects 
and application of the method are more the concern of this 
thesis. This chapter includes historical background of the 
method, a review of the Lagrange multiplier and penalty 
function methods, and a development of the algorithm for 
equality and inequality constrained problems. Rigorous 
mathematical development of the method is given by Hestenes 
[ 6 ]. 



A . BACKGROUND 

The multiplier method was independently introduced in 
1968 by Hestenes [7] and Powell [8]. Both developed an 
augmented Lagrange function and solved a series of uncon- 
strained minimizations followed by a simple update of the 
multiplier vector. Powell showed that, if the function had 
continuous second derivatives, the method would converge 
locally at a linear rate while the penalty parameter 
remained finite. As will be shown later, this eliminated 
the inherent numerical ill-conditioning of classical penalty 
function methods. 
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Extensive testing and modification has been done to the 
method of Hestenes and Powell. Miele et. al. [9,10] applied 
the method extensively to equality constrained problems. 
Tripathi and Navandra [11] experimented with updating the 
multiplier after each one dimensional search instead of after 
each unconstrained minimization. Rockafellar [12,13] expanded 
the method to nonlinear inequality constrained problems. 

Global convergence of the method was proved by Rockafellar 
[13] for convex programming problems. Bertsekas [14,15] 
provides convergence proofs for the general nonlinear problem. 



B . THE LAGRANGIAN 

The general nonlinear equality constrained problem is 
defined: 

Minimize f(X) (2.1a) 

subject to h^ (X) = 0 (i=l , . . . ,m<n) (2.1b) 

The theorem associated with Lagrange multiplier method states 
that : 

"If X* affords a local minimum to f(X) subject to the 
constraints h^(X)=0, then there exists a unique set of 



multipliers. A., (i-l,....,m) such that if 

m 

L(X,A) = f(X) + E A.h. (X) , 

i=l 



then 



m 



VL(X*,A*) = Vf(X*) + 5Z1 A. Vh. (X*) = 0 

i=l ^ ^ 



and 



L"(X*,A*) = 



9 L(X*,A^) ^ 0 



(2.2a) 



(2.2b) 



(2.2c) 
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where V denotes the gradient of the function and L" 
denotes the second derivative of L" [6] . 

Equations 2.2b and 2.2c are the necessary conditions for 
locally constrained minima. Equation 2.2b and the feasibili- 
ty condition (Eq. 2.1b) constitute the Kuhn-Tucker necessary 
conditions for optimality [6]. It is assumed that f (X) and 
h^(X) are second order differentiable and that the gradients 

Vh. (X) are not zero at X*. 

1 

The problem can now be stated in terms of the equivalent 
classical Lagrangian. 

Minimize L(X,X) (2.3a) 

subject to h^ (X) = 0 ( i=l , . . . ,m<n) (2.3b) 

Assuming the existence of the saddle points of the Lagrangian 
L(X,X), the condition exists: 

L(X*,A) < L(X*,X*) < L(X, X*) (2.4) 

The optimal pair (X*,A*) can be obtained by first minimizing 
L(X,A) respect to X, then maximizing L(X^,A) with respect to 
A by update equation, 

= A^ + c[h. (X^) ] (2.5) 

where c is a scalar parameter (stepsize) , k is the iteration 
number, and X^ is the local minimum of L(X^,A^). The procedure 
is repeated until convergence is attained. This is the so 
called "primal-dual" method. 

Serious disadvantages are encountered in the primal-dual 
method. First, the problem (Eq. 2.3) must have a locally 
convex structure for the dual problem to be well defined and 
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for Equation 2.5 to be meaningful [16]. Second, a large 
number of iterations are usually required to minimize L(X,X) 
(Eq. 2.2a) since the ascent iteration (Eq. 2.5) converges only 
moderately fast. Thus, primal-dual methods have found 
application in only a limited class of problems where minimi- 
zation of the Lagrangian (Eq. 2.2a) can be efficiently carried 
out due to special structure, as shown by Luenberger [16], or 
where the design problem exhibits a unique form, as shown by 
Schmit and Fleury [17]. 

C. PENALTY FUNCTION f4ETH0DS 

Penalty function methods have been used extensively since 
the mid-1940's [18]. They are considered to be efficient for 
inequality constrained problems. 

Given the nonlinear inequality constrained problem: 

Minimize f(X) (2.6a) 

subject to g^ (X) ^ 0 (i=l,...,2.) (2.6b) 

The general exterior penalty function is defined: 

I 

F^(X,c) = f(X) + c V^, <{>(t) (2.7) 

® i=l 

where cj)(t) is some scalar penalty function of the constraints 

and c is some scalar penalty parameter. The most common 

2 

penalty function is the quadratic (J)(t)=t /2. However, it may 
be desirable at times to use other penalty functions. In this 
study the quadratic penalty function is used such that 
Equation 2 . 7 becomes 



12 



( 2 . 8 ) 



F^{X,c) = f(X) + I C Pi(^) 

i=l 

fg. (X) , if g. (X) ^ 0 

where P^(X) =< ^ 

^ 0 , otherwise 

The problem is now an unconstrained minimization of F^(X). 

The convergence of the method is easily seen by a simple 
numerical example. 

Example 1 [2]. Minimize x such that 1-x < 0. 

Solution: First, examine the solution to the problem 

by primal-dual method. Equation 2.2a becomes 

L(x,X) = X + A(l-x) (2.9) 

where A is a real non-negative number. From the stationary 
conditions of Equation 2.9: 

(x*,A*) = (1,1) 



Solving by exterior penalty method. Equation 2.8 becomes 
F (x,c) = X + ^ p^ (x) 



( 2 . 10 ) 



where p(x) = 




X > 1 

otherwise 



From the stationary conditions, a minimum exists at 

X- = 1 - 1 

c 

To obtain the optimal x*, a series of unconstrained minimiza- 
tions are solved while increasing the penalty parameter c 
toward infinity. It is apparent that as c->-“, x'->-x*. No exact 
solution is obtained and the optimum is approached from the 
infeasible region. 
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The plots of Fg(x) vs. X and f(x) vs. c are given in Figs. 1 
and 2, respectively. Note in Fig. 1, as c gets larger, the 
function changes more rapidly and the optimum becomes more 
difficult to find regardless of the minimization technique. 
This is a cause of niimerical ill-conditioning inherent with 
penalty methods. Figure 2 shows the asymtotic convergence 
of the method. 

Interior penalty functions have the advantage of 
approaching the optimum from the feasible region thus 
yielding a feasible solution. However, the penalty function 
is discontinuous at the constraint boundaries. Also, the 
same problems of ill-conditioning and slow convergence exist 
as seen by Example 2 . 

Example 2. Solve Example 1 by the interior penalty 
function method. 

Solution: One form of the interior penalty function 



xs 



m 






- I r. 7 

X=1 



(X) 



( 2 . 11 ) 



Substituting , 



F. . (x) = - - 

int c 1-x 



( 2 . 12 ) 



It can be shown analytically that a minimum exists at 



X' = 1 + i 

It is again apparent that x'^x* only as c-^-™. 

Figures 3 and 4 illustrate that the same problems of ill- 
conditioning and slow convergence exist as with the exterior 
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Fig. 2. Convergence of Exterior Penalty Function 
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penalty method. An exact solution is again not possible but 
the solution achieved here is, however, feasible. 



Extended interior penalty methods [19] avoid function 
discontinuity at g^(X) =0 inherent in the interior penalty 



method. It is, therefore, less susceptible to ill-conditioning. 
It does exhibit the same slow convergence of other penalty 
methods due to the requirement to increase c to infinity. 

It is for these reasons that the multiplier method is an 
attractive alternative. 

D. THE EQUALITY CONSTRAINED PROBLEM 

The multiplier method can be perceived to be a combined 
primal-dual and penalty function methods. Though they are 
theoretically similar, their behavior is quite different. 

It has been shown that the original equality constrained 
problem (Eq. 2.1) is equivalent to the classical Lagrangian 
(Eq. 2.3). Since Equation 2.3 is still an equality constrained 
problem, it can be solved by the usual exterior penalty 
function method. The quadratic penalty function is used so 
first derivatives are continuous. Substituting Equation 2.3 
into Equation 2.8: 




i=l i=l 



(2.13) 
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where P£(X) = h^(X) 

Equation 2.13 is defined to be the "Augmented Lagrange 
function" for the equality constrained problem. 

By the nature of penalty function methods, as c goes to 
infinity. Equation 2.13 converges to the solution of Equation 
2.8. Concurrently, by choosing an appropriate value of X 
with a suitable update formula. Equation 2.13 can be solved 
by a series of unconstrained minimizations to obtain a solu- 
tion to the original problem (Eq. 2.1). 

The selection of X can be significant to the behavior of 
the function as seen in the following two extreme cases. 

First, take X^=0 for all unconstrained minimizations. 

Equation 2.13 becomes 

A(X,X,c) = f(X) + § ZZ hf(x) (2.14) 

^ i=l 

which is the usual quadratic exterior penalty function 
(Eq. 2.8). It has previously been shown that the function 
only converges to a minimum f(X*) as c goes to infinity. It 
has also been shown to be slow in converging, susceptible 
to numerical ill-conditioning, and to attain only a near 
optimum solution from the infeasible region. 

Next, consider the case where At the minimum, 

the stationary condition requires that 
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m 

VA(X*,X*,c) = Vf(X*) + YZ >^-h.(X*) 

i=l ^ ^ 

m 

= £ ^ h. (X*)Vh(X*) = 0 (2.15) 

i=l ^ 



The feasibility condition hj^(X*)=0 implies that Equation 2.13 
is independent of the value of c. This leads to two important 
results. First, if the optimum A* is known initially, the 
solution can be obtained in one unconstrained minimization. 
Second, since A(X,A,c) is independent of c at the optimum, it 
is not necessary to sequentially increase c to infinity to 
attain a solution. The second result implies that a finite 
c can be chosen, thus avoiding the inherent ill-conditioning 
of the penalty methods. 

The task is now to find initial values for c and A with 

sequential update formulas for each to achieve a suitable 

rate of convergence. Collecting terms. Equation 2.15 becomes 
_ m 

Vf(X*) + C [A*+ch.(X*)] Vh. (X*) = 0 (2.16) 

i=l ^ ^ ^ 

By the Kuhn-Tucker conditions, at the optimum (X*,A*). 

m 



Vf(X*) + C A* Vh. (X*) = 0 

1 1 

1=1 



(2.17) 



Thus, Equation 2.16 reduces to Equation 2.17 in the limit. 
This implies as update formula for A such that 



= A^ + ch. (X*^) (2.18) 

1 1 1 

~ Ic th 

where X is the solution to the k unconstrained minimization. 
Proposed initially by Hestenes [6], Equation 2.18 remains the 
most popular update formula for A. 



19 



Little experimentation has been done with choosing and 
initial X°. As shown earlier, if X°=X* the solution is 
obtained in one unconstrained minimization. Obviously, the 
closer X*^ is to X*, the more rapid the convergence. A widely 
accepted practice is to choose X?=0 due to the computational 
convenience, and because no other multiplier has consistently 
proven more efficient. Active and violated constraints are 
immediately identified in this case since a non-zero X^ can 
only occur for an active or violated constraint. A constraint 
is active if the multiplier is at its optimum X^t^O when X-»X*. 
This eliminates any extra computations to check constraint 
behavior. 

Before determining the choice of the initial penalty 
parameter and the update formula for c, it is first con- 
venient to examine the convergence of the method which is 
directly related to the choice of c. Various proofs of 
linear convergence to a local minimum have been developed 
[8,20,21]. Rockafellar [13] proved global convergence of the 
method for convex programming. Bertsekas [14,15] provides 
a rigorous proof of the method's convergence for the general 
nonlinear problem. He compares the convergence of the 
various multiplier methods, penalty methods, and primal-dual 
methods [22]. His results for the Hestenes' [6] multiplier 
method are siammarized here. 

Recall the assumptions made for a local minimiam to exist: 

-k 

Assumption 1. There exists a local minimizing point X 

of problem 2.1 which satisfies the second 
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order sufficiency conditions for an isolated 
local minimum, i.e., f and are twice 
continuously differentiable in a neighbor- 
hood of X^, the gradients Ah^ (X^) , (i=l,...,m) 

are linearly independent and there exists a 
Lagrange multiplier vector A, such that 
VLU^,X^)=0 and ] V^L (X^, X ^ | >0 [22]. 

Assumption 2. The penalty function 4>(t) is twice continuously 

dif ferentable in an open interval containing 
zero and 4>” (0) >0, where <})" denotes the second 
derivative [22], 

Also, assuming the Hessian matrices V f(X), V hj^(X), and the 
second derivative 4>" are Lipschitz continuous,^ there exists 
a scalar c*>0, and M>0 such that for every c^>c* the function 
L^(X,X) has a unique minimum X^^(X,c). Furthermore, 





1 1"’^ 

Y* — 


X* 


II < |x 




(2.19) 




1 1 ^ 


1 1 


c 


and 


1 1 


_ T*l 1 < ^ 


x^-x* 1 1 


(2.20) 






C 


where = 


X^ 

1 


+ c h. (X^) 
1 




The 


notation | 


M 


1 denotes 


the usual Euclidean norm. 






From Equation 


2.20, if 


Ic • 

c ^c<“ for the non-decreasing 


posi- 



tive penalty sequence {c^^ so as to ensure M/c^-^1, then the 
sequence {X^} converges to X linearly. If c^ goes to infinity. 



A function is said to satisfy a Lipschitz condition of 
order m on the closed interval (a,b) if there exists a constant 
c such that | f (x^ ) -f (x^ ) | <c | x^-x^ | for all values ,x^ on (a,b) . 
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Ic ““k. 

M/c ^0, and the sequence {A } converges superlinearly . From 

Equation 2.19, for sufficiently large but finite c' , the 

sequence converges to X* since {x^} converges. If c^ 

goes to infinity, {X } converges by the same argument. 

”*k 

The rate of convergence of {X } is represented by the right 

**k 

hand side of Equation 2.19. Since (A } converges at least 
linearly, {X } converges at least linearly. This has been 
shown to be significantly faster than pure penalty or primal- 
dual methods. Numerical ill-conditioning is avoided since 
this linear convergence rate is achieved with a sufficiently 
large finite c. 

The convergence criterion of Equations 2.19 and 2.20 are 

global in nature since no bounds restriction has been imposed. 

No restrictions on the convex or non-convex nature of the 

problem are specified as well. The global convergence of the 

multiplier method is contingent upon the ability of the un- 

f “* k \ 

constrained minimization method to generate a sequence lX j 

“ k k 

which are well defined local minimums to the function A(X ,A‘ ,c). 
Naturally, the function A(X' ,A ,c) may have other local mini- 
miams to which the unconstrained minimization method may be 
attracted. Unless the unconstrained minimization method 
stays in the neighborhood of the same local minimum, the con- 
vergence argxament is invalid and there is no assurance that 
the multiplier method will perform any better or worse than 
the penalty methods. It should be noted that the usual 
practice is to use the last point X‘ of the k minimization 
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as the starting point of the (k+l)th minimization. This 
generally produces sequences {x } which are close to the 
same local minimum of A(X,A,c). 

The convergence rate of the multiplier iteration (Eq. 2.18) 
is linear with the convergence ratio essentially inversely 
proportional to the penalty parameter c [22]. This fact is 
strongly dependent on Assumptions 1 and 2. If either of the 
assumptions is relaxed, the convergence rate may become 
sublinear or superlinear as the following examples show. 

Example 3 [22]. Consider the scalar problem minimize 

2 

X /2 such that ^=0 with an optimal solution ^*= 0 , A*=0. In 

this example Assumption 2 is not satisfied. For A^O, 

2 

4)(t) = |t| /3, and c=l, Equation 2.13 becomes 



1 



X 



(A,l) 



-1-/(1-4A) 

2 



Starting at A°=0, Equation 2.18 becomes 
,k+l ^ 1- 



>1 1 

■4A ) 



The lim,A 
►-00 



k.+l 



k->o" ^ 



Ak 



1, i.e., a sublinear convergence rate. 



Example 4 [22]. Consider Example 3 where (|) (t)=2/| 1 1/3, c=l. 
Again, Assvimption 2 is not satisfied. The solution to Equation 
2.13 becomes 



4 
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starting at A°=0 , Equation 2.18 becomes 



^k+1 ^ ^ (-1 + /l-4A^^) 

It can be shown that 



lim 

k->oo 



I A 



k+1 






1 



hence, a super linear convergence (order 2) . 

Example 5 [22], Consider the problem, minimize lx|^/2 

such that x=0. Again x*=A*=0, but Assumption 1 is not satis- 

2 

fled. For 4>(t)=t /2, c=l, the solution to Equation 2.13 
becomes 



x^(A,l) = 
and Equation 2.18 
= X>= 



+ 



(-l+/T^) 

2 



becomes 

(-1 + /l-4A^) 
2 



Again , 



lim |A^"^| _ , 

k^-oo k 2 ^ 

(A^) 

hence, superlinear convergence. 

It can be shown that the convergence rate is influenced 
substantially by the rates of change of the derivatives 
(curvature) of the primal function p (u) = min f(X), where 
u = h(X), and the penalized primal function derivatives 
p (u) +c [({) (u) ] near u=0. The convergence is faster if the rate 
of change of Vp(u) is small and the rate of change of 
c[V(()(u)] is large near u=0. 
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In Example 3, the rate of change of V<j) (u) is small near 
u=0 and convergence is slow, while in Example 4, it is 
large and convergence is rapid. In Example 5, the rate of 
change of Vp(u) is small near u=0, thus the fast convergence. 

The following example shows that in the absence of 
Assumption 1, Equation 2.18 may not lead to convergence for 
any c>0 when <p is essentially quadratic. 

Example 6 [22] . Consider the problem, minimize - | x | 
such that x=0 where Kp<2. For any c>0, there exists a neigh- 
borhood of X =0 such that the augmented Lagrangian (Eq. 2.14) 
does not have a local minimum for any X when <p is essentially 
quadratic. This can be corrected by using (J)(t) = |tl^ + t /2 

t 

where l<p <p. 

The two extreme cases are now examined. First, suppose 
k. _ 

"X = X*, Equation 2.19 becomes 
I 1 X‘^ - X* 11^ 0 

_ k — 

The norm is always non-negative; therefore, X' = X* and the 
solution is reached in one unconstrained minimization. 

Next, letting X^ = 0 for all k, i.e. , the penalty function 
method. Equation 2.19 becomes 



“ k ” k 

In this case, X ^X* only if c -»■<», requiring many unconstrained 
minimizations and a sublinear rate of convergence. 

It has been shown that the rate of convergence is directly 
dependent on the penalty parameter, c. The convergence 
estimates (Eqs. 2.19, 2.20) are valid for c greater than some 
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threshold value, c* which depends on (X) and the problem 

data. In general, c* is unavailable and it is impossible to 

know apriori the range of values of c for which Equations 

2.19 and 2.20 are valid and imply fast convergence. 

A penalty parameter update sequence is required to increase 

c monotonically with each unconstrained minimization. As 

c^°°. Equations 2.19 and 2.20 will eventually become valid. 

It should be noted that large values of c can induce ill- 

conditioning, making the unconstrained minimization of A(X,X,c) 

difficult. On the other hand. Equations 2.19 and 2.20 indicate 

-k - 

faster convergence of {X } to X* for large values of c. 

An update sequence is recommended whereby c is multiplied 

k+i k 

by some constant Y>lf i.e., c = yc . The penalty parameter 



c is increased in this manner to some significantly large c 



max 



Bertsekas [22] recommends y not much larger than 1 to avoid 
ill-conditioning effects in the first few unconstrained mini- 
mization iterations. This update scheme will be used since 
it has been subject to the most testing in recent years. 

The choice of an initial c requires more experimentation and 
will be discussed later. 

Other methods are available for updating the penalty parameter 
c. Powell [7] suggests multiplying c by some constant 6>1 
only if the violated constraint, as measured by ] | h [X ( A , c) ] | [ , 
is not decreased by a certain factor over the previous mini- 
mization, i.e., c^'*’^ = 6c^ if ] |h[X(A^',c^) ] | | > Yl|h[X(A^ 1 1 

and c^ ^ = c^ otherwise, where 6>1, and, y< 1 ^re some specified 
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scalars. This scheme generates a penalty parameter sequence 
that will be constant after a certain index and will achieve 
convergence by virtue of enforcing the asymtotic feasibility 
of the constraints, i.e., 

lim 1 I h [X ( c^) ] I I = 0 

Another similar update scheme is to use a penalty parameter 
vector such that Cj^ is updated by the Powell method only if 
the constraint hj^[X(A,c)] is violated. This case of a 
separate penalty factor for each constraint corresponds to 
m.erely scaling of the constraints. A simple modification to 
Equations 2.19 and 2.20 is used to prove convergence of 
this case. 

Finally, define a dual function A^(A) for the augmented 
Lagrange function A(X,A,c) such that 

A (A) = min A(X,A,c) (2.21) 

C X 

Ic *“ 

It can be shown that if X (A,c) is the solution to Equation 
2.21, then A^(A) is a twice continuously differentiable 
convex function with gradient given by 

V^A (A) = Vh[X^(A,c)] (2,22) 

A C 

where h[^(A,c)] is the constraint vector (Eq. 2.1b) evaluated 
at X^. Substituting Equation 2.22, Equation 2.18 becomes 

X^+l ^ ^ v^A^(A) (2.23) 

This relation shows that the multiplier iteration (Eq. 2.18) 
is an iteration of steepest ascent for finding the maximum 
of the dual function, ( A) . Equation 2.23 is equivalent to 
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a steepest ascent iteration for quadratic penalty functions, 

2 

4>(t) = t /2. The multiplier method can thus be viewed as a 
primal-dual method with a limited search for the optimal 
Lagrange multipliers in the dual space. 

The Hestenes ' multiplier method for nonlinear equality 
constrained problems will not be summarized. Quadratic 
penalty functions are used in this algorithm. 

Step 1 : Select A°=0 and an appropriate penalty param- 

eter c*^>0. Set k = 1. 

Step 2: Solve min A(X, X ,c) , defined by Equation 

2.13. Denote the solution X . 

Step 3: Update X by Equation 2.18. 

— k+l — k -k — k 

Step 4: If X = X , stop. (X ^X'^) is the optimal 

solution. Otherwise, go to Step 5. 



Step 5: Set X^ = X^"*"^. 



If c <c , where c is 
max max 



some significantly large number, update c 
k+l k 

by c = yc , where y is some increase 

k 

factor greater than one. Otherwise, c =c 



max 



Set k=k+l. Go to Step 2. 

Note that no assumptions or restrictions have been made 
concerning the nature of the objective or constraint function. 
If Assiimptions 1 and 2 are satisfied, the solution will 
converge to a global minimum; otherwise, only a local minimum 
can be guaranteed. 
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E. THE INEQUALITY CONSTRAINED PROBLEM 



The multiplier method has been shown to be theoretically 
attractive for equality constrained minimization problems. 

In reality, many engineering problems involve inequality 
constraints. It is, therefore, necessary to extend the 
discussion of the multiplier method to include inequality 
constraints . 

Consider the nonlinear inequality constrained problem: 

Minimize f(X) (2.24a) 

such that g^(X) - 0 (i=l,...,£) (2.24b) 

where X is the vector of n design variables. Introducing 
slack variables. Equation 2.24b becomes 

g^(X) + z? = 0 (2.25) 

2 th 

where zf is the slack variable for the i^ constraint. The 

problem is now an equality constrained problem of the form 

of Equation 2.1; however, the number of design variables has 

increased to n+Z. The augmented Lagrangian (Eq. 2.13) becomes 

A(X,Z,A,c) = f (X) + t 

i=l ^ ^ 

i, , _ ^2 

+ I (gj_(X) + x^) (2.26) 

i=l 

If the number of constraints, Z is much greater than the 
number of design variables, n, as is often the case in 
engineering design problems, the unconstrained minimization 
problem becomes sizable. The scope of the problem can, 

2 

however, be reduced by eliminating the slack variables, 
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The unconstrained function, A(X,Z,X,c) is first minimized 
with respect to Z. For a local minimum to exist, the station- 
ary conditions 
3A 



3z, 



= 0 (i=l , . . . ,n) 



must hold. Differentiating A(X,Z,A,c) 

A — 7 

jj- = 2X.X. c[g. (X) zp (2z^) = 0 

^i - 2 

^i^~ ^i^^^ + z^] = 0 

The solution to Equation 2.29 is 



(2.27) 

(2.28) 
(2.29) 



z. = 0 



or 



2 

'i 



A. 

1 



z7 = ^ - g^_(x) 



Since zf < 0 is meaningless, the solution becomes 

2 _ 
zf = max [0, -gj_(X) - -^] 



(2.30) 



Equation 2.30 shows that z^ is no longer an independent varia- 
ble. From this equation it is observed that if gj^(X) is a 

critical constraint, z. = 0. If g. (X) is non-critical , z. > 0. 

1 1 

Therefore , 

g. (X) zf = max [g. (X) , (2.31) 

with the slack variables eliminated, the augmented Lagrangian 
becomes 



A 



o 

(X,A,c) = f(X) -t- C + f 

i=l 



^i. 



where = max [g^(X), — 



(2.32) 
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This is referred to as Rockafellar ' s augmented Lagrange 
function. Note that A(X,X,c) has continuous first derivatives 
with respect to c, but discontinuous second derivatives at 



methods, i.e., Newton's method, for unconstrained minimization. 

From Equation 2.31, Hestenes' update formula for X for 
the inequality constrained problem becomes 



The algorithm for Hestenes' multiplier method is easily modi- 
fied to include inequality constraints by adding Equations 
2.32 and 2.33 to steps 2 and 3, respectively. Note that 
since the inequality constrained problem is transformed to 
an equivalent equality constrained problem, the convergence 
properties are identical. 

Example 1 can now be solved by the multiplier method. 
Example 7 [2] . Minimize x such that l-x£0. 

Solution: Equation 2.32 becomes 



gj^ (X) = -X/c. This preludes the use of second order 



k+1 



= max [0, X^ + c g^ (X^) ] 



(2.33) 



A(x,X,c) = X + c x) ^ + ^(1-x), if 1-x-O 



X + c 




, otherwise 



otherwise 



Applying Hestenes' algorithm with X'^=0 and c*^=l, the augmented 
Lagrangian becomes 
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This has a solution x°=0. Updating A by Equation 2.32, 

A^ = max [0 , A°+cg (x'^) ] 

= max [0, 0 + 1(1)] = 1 

With A^=l and c^=l, the next unconstrained minimization of 
A(x,A,c) becomes 

A^(x,l,l) = X + j (l-x)^/2 + (1-x) , if 1-x > 0 

- 1/2 , otherwise 

The solution becomes x' =1. Updating A, 

A = max [0 , A + c g (x ) ] 

= max [0, 1 + 1(0)] = 1 
1 

Since A =A'', stop the calculation. The optimal solution is 
(x*,A*) = (1,1). 

Note the convergence of the method even for a constant penalty 
parameter c. 

The functions A° and A^ are plotted in Fig. 5, and the 
objective value vs. c is plotted in Fig. 6. Several facts 
are evident from this example. 

1. A solution is obtained in a few unconstrained minimi- 
zations . 

2. Each unconstrained problem is a smooth curve. Thus 
ill-conditioning is avoided. 

3. Convergence is not asymtotic and an exact solution is 
attainable. 

4. With an initial Lagrange multiplier of zero, the 
solution is obtained from the infeasible region. 
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Fig. 5. Augmented Lagrangian Function 
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5. The optimal Lagrange multiplier is obtained. 

6. Any starting point is possible. 



The method, therefore, shows attractive theoretical features 
over pure penalty or primal-dual methods. 

The multiplier method algorithm is now summarized for the 
nonlinear equality and inequality constrained problem: 
Minimize f(X) 

such that g^{X) < 0 (i=l,...,5,) 

and hj (X) = 0 ( j= H+1 , . . . , 5,+m) 

Algorithm: 

Step 1. Choose an initial Lagrange multiplier = 0 and 
an initial penalty parameter c° > 0. 

Step 2. Solve the unconstrained minimization problem 
Min A(X,A,c) = f(X) 



m ^ 

+ C + 1 1 

1=1 

m+£ - 

+ C (X) + I h^(X) ] 

j = £+l ^ 



A. 

where = max [g^(X), — 

Step 3. Update the Lagrange multipliers, A by 



A^"*"^ = A^ + c max[g. (X^), — (i=l,...,5.) 
X. 1 1C 

Aj'^^ = Aj + c hj(X^) (j=m+l, . . . ,m+£) 

— k + 1 

step 4. If A = A , stop. The optimal solution is 
(X*,A*) = (3^,A^). Otherwise, go to Step 5. 



34 



step 5 . 



— k+1 

Set \ = X . Update the penalty parameter c by 

k+1 k 



c = yc . 

^k+1 ^ 4 . k 1 4 . k k+1. 

If c > c , set c = c , else set c = c 
max max 

Set k = k+1; go to Step 2. 

This is the theoretical algorithm used in the multiplier 
method. Minor variations are described in the next chapter 
which are designed to improve the computational efficiency, 
reliability, and accuracy of the method. 
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III. COMPUTATIONAL ASPECTS 



The effectiveness of the multiplier method is dependent 
upon the accuracy of the unconstrained minimization, choice 
of Lagrange multiplier update formula, penalty parameter, 
the form of the penalty function, and the objective and 
constraint functions themselves. The theoretical effect of 
the Lagrange multiplier and penalty parameters on the method's 
convergence rate was described in Chapter II. The require- 
ment that the objective and the constraint functions satisfy 
Assumptions 1 and 2, listed in Chapter II. D, has also been 
established. 

This chapter will examine the effects of the Lagrange 
multiplier and penalty parameter from a computational view- 
point. As stated earlier, the penalty function is quadratic 
throughout this discussion. The computational aspects of 
the unconstrained minimization program and associated one- 
dimensional search subprograms are first examined. 

A. THE UNCONSTRAINED MINIMIZATION PROBLEM 

A sequence of unconstrained minimizations is required in 
obtaining a solution by the multiplier method. Of the many 
unconstrained minimization techniques available, the Davidon- 
Fletcher-Powell [DFP] method [1] was selected because of its 
accuracy and rapid convergence. Figure 7 compares the DFP 
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COMPARISON OF UNCONSTRAINED MINIMIZATION METHODS 




ITERATION 

Fig. 7. Comparison of Unconstrained Minimization Methods 
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method to the methods of steepest descent, Fletcher-Reeves 
conjugate gradient method developed as part of this study, 
and the Fletcher-Reeves method contained in the CONMIN 
program for the extremely ill-conditioned "banana" function. 
Note the DFP method converges in fewer iterations than the 
other methods with little or no loss of accuracy. 

To solve the general unconstrained minimization problem, 
it is necessary to first calculate a search direction, S, 
initially the direction of steepest descent. A one-dimensional 
search is then used to find the minimum of the objective 
function in the search direction, S. The one-dimensional 
search computes a scalar, which minimizes the function 
f(Xj^_j^ + . The new X-vector associated with the function 

minimum becomes 



\ \-l 

A new search direction is calculated, and the iteration is 
repeated until a specified convergence criteria is satisfied. 

The algorithm for the DFP method is described as follows 
for the general unconstrained minimization problem: 

Step 1. Select an initial point X^ and an initial posi- 
tive definite symmetric matrix, = I (identity 
matrix) . Set k=l. 

Step 2. Calculate the gradient, Vf(X^), and set 

S = - H Vf(X^) 
o o o 

step 3. Compute Xj^ = + ot* Sj^ 
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step 4 . 



where a* minimizes f + aSj^) , i.e., a* is 

the solution to the one-dimensional search. 
Computer ^ 

where AF = Vf(Xj^) - Vf(Xj^_j^) 



\ - \-l = 

r ^ AXAX'^ 

^ Ax"^AF 

S _ (HAF) (HAF)'^ 

~ =T C 

AF HAF 

Step 5: Compute , set k=k+l, go to Step 3 

The basic algorithm is modified to provide (1) scaling of the 
variables, (2) appropriate update and resets, and (3) a suita 
ble termination criterion. 

A feature is added which normalizes the search vector 
such that 



"NORM 



k ' max 



For excessively ill-conditioned problems, pre-scaling the 

variables may also be required. 

Resets and updates of the vector, S and the matrix, H, 

are incorporated to prevent method breakdown from round-off 

error and other instabilities. The positive definiteness 

of H is preserved in theory only if a* provides a true 

— T — 

minimum point, i.e., = 0. The update of H is, 

therefore, skipped if e , and the update 
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was not skipped on the previous iteration. A tolerance 
value, e = 0.01 was co-putationally found to provide a good 
update criteria with negligible effect on the convergence 
side. 

Two checks were made on S to insure a valid search 
direction. Note first that if H=I, the search direction is 
that of steepest descent. Now at iteration k, if Sj^* Vfj^ (X) >0 , 
the search direction will not reduce the objective and a new 
search direction is found. Second, if at the end of the 
one-dimensional search f >f (Xj^) , the function is ob- 

viously not a minimum and a new search direction is found. 

In each case, the H-matrix is reset to identify, I, and a 
search in the direction of steepest descent is performed. 
Finally, to maintain stability of the H-matrix, it is reset 
to the identify matrix every NDV+1 iterations, where NDV is 
the number of design variables. 

The method is terminated when the relative or absolute 
difference of the objective function is less than 0.001 for 
two consecutive iterations or when a preassigned maximum 
number of iterations is exceeded. The somewhat strict 
convergence criterion was chosen since the performance of 
the multiplier method is highly dependent on an accurate 
unconstrained minimization. 

The DFP method has been shown to be a reliable first 
order method with quadratic convergence. It has one 
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additional feature in that it provides a good approximation 
to the Hessian matrix. 



H = 



(X) 

-2 

3X 



Thus, the m.ethod provides many of the advantages of a second 
order, Newton type method, without the tedious calculation 
of second derivatives. 

Finally, the DFP method requires subprograms to calculate 
gradients and perform the one-dimensional search. Gradient 
calculation was done by the first forward finite difference 
method. While a central difference method would improve the 
accuracy of the calculations, the first forward finite 
difference method with a step size of 0.01 has provided 
acceptable results in experimentation thus far. Discussion 
of the one-dimensional search subprogram follows in the next 
section. 



B. THE ONE-DIMENSIONAL SEARCH 

A combination of the Golden Section method and polynomial 
interpolation was used for the one-dimensional search. The 
one-dimensional search calculates a scalar, which minimizes 
the function, f (Xj^_^ + , It has been shown that the 

accuracy of computing is critical to the performance of 
the unconstrained minimization subprogram. The Golden Section 
method was used because of its reliability and accuracy. The 
method is, however, less efficient than polynomial interpola- 
tion methods in many cases , The Golden Section method 
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converges linearly while polynomial methods have no guaranteed 
rate of convergence. It may also require many function evalua- 
tions per iteration while polynomial methods require few. 

The Golden Section method is based on a Fibonacci search 
[16] which yields a sequence of intervals of uncertainty whose 
widths tend to zero faster than any other method. Given upper 
and lower bounds, and where x denotes the scalar, a, 

in Equation 3.1, two interior points, x^ and X 2 are found such 
that 

u 2 1 il 

and the ratio 

X- 

— = (1 + /5)/2 = 1.61803 
^1 

This is the Golden Section number. The interior points, x^^ 
and X 2 are defined as 

Xi = (1 - T)Xj. + TX^ 

Xj = TX^ + (1 - T)X^ 

where x = (3-/5) /2 = 0.38197 

The bounds are revised by a simple update routine and one new 
interior point is calculated at each step. The process is 
continued until a given relative convergence tolerance, e, 
is satisfied. The algorithm for the Golden Section search 
is given in Appendix A. 

The relative convergence tolerance, e, is defined 
e = Ax/(x^ - x^) 
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Recognizing that the interval is reduced by the fraction, t 
(38 percent) each iteration, the maximum number of function 

evaluations, N per iteration is calculated 

,, ,N-3 

e = (1-x) 

or N = In ( e) /In (1-T ) + 3 

= -2.078 ln(e) + 3 

Note that by defining e, a fixed number of function evalua- 
tions are performed each iteration. If the absolute conver- 
gence tolerance. Ax were specified, the number of function 
evaluations will vary each iteration. Experiments have 
shown an absolute convergence criteria to yield slightly 
better accuracy with nearly the same number of function 
evaluations. Therefore, an absolute convergence criteria 
was specified to give the same or better efficiency than 
the relative criteria and with improved accuracy. 

The Golden Section method requires a logical guess to 

the first x such that x and x„ bracket the minimum, 
u u Z 

Selection of x^ was performed such that if more than two 
guesses were required, the last three points chosen would 
yield the values x^, and x^. This provided a smaller 

initial bound than if the original x^ was used and also 
saved one function evaluation since x^^ was already found 
for future calculations. 

Finally, a cubic approximation was performed using the 
last four points from the Golden Section method. A better 
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minimum was found in almost every case at the cost of only 
one additional function evaluation. 

C. THE MULTIPLIER METHOD 

Experimentation with the multiplier method involved in- 
vestigating the effects of the penalty parameter, c, the 
update factor, y, and the update formula for the multipliers, 
A. It was desired to compute these parameters internally if 
possible, such that the method's performance would be inde- 
pendent of the problem. This is necessary to provide a 
method which can be used simply and reliably for many 
engineering applications. Effects of scaling, variable 
bounds, and perturbed constraints were not investigated but 
will be discussed briefly for completeness. 

The choice of the initial penalty parameter, c°, can 
have significant effect on the efficiency of the multiplier 
method. No universally acceptable method has been found to 
select an initial penalty parameter, c°. Two methods have 
been found to provide good initial estimates and are recom- 
mended here: (1) choose the initial penalty parameter, c°, 

of the same order ot magnitude as the initial objective 
function, or (2) choose the initial penalty parameter such 
that I I I second method requires the gradient 

of each constraint which is not directly available in the 
current program. Both methods have provided acceptable 
results for the examples tested. They cannot guarantee a 
reasonable convergence or a reliable answer in all cases. 
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Bertsekas [22] recommends as increase factor, y not much 
greater than 1, A y=2 worked well in all the cases tested. 
Reasonable success was achieved with y=5, but caution should 
be used since instability and ill-conditioning occurred when 
c*^ was chosen too large. Bertsekas' recommendation of y>l 
to a maximum of y=2 appears to be adequate in most cases. 

The maximum penalty parameter, c is not as critical 

to problem stability as the initial c*^. It was noted that 

ill-conditioning does occur for too large. A good 

guideline is to choose c of order five to six times that 

of c'^. If ill-conditioning occurs, it will be necessary to 

lower c , c*^ or both, 
max 

Choosing an initial set of multipliers, other than 
zero was not investigated. The convenience of identifying 
active and violated constraints without additional computa- 
tions when X?=0 tends to overshadow any computational advan- 
tage which may be attained by choosing a non-zero initial 
multiplier. 

Normalization of the constraints is necessary in the 
function subprogram to avoid breakdown of the multiplier 
method. Scaling may be required for extremely ill-conditioning 
problems. Various scaling methods exist, all of which 
provide some means making the variables, objective, and 
constraints (or their gradients) equal to or near the same 
order of magnitude. 
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At present, the algorithm treats all side constraints 
or boxmds as additional problem constraints. Treating side 
constraints directly can greatly improve the computational 
efficiency, as well as simplifying problem programming. 
Ragsdell [23] has shown success in treating variable bounds 
(side constraints) directly but does not offer any theoretical 
justification. Further research on direct treatment of side 
constraints is required. 

The multiplier method often converges computationally to 
a solution that is slightly infeasible. This may not be 
desirable from the practical point of view. Imai [2] con- 
sidered the following perturbed problem: 

Minimize f(X,X) 

such that g^(X,X) i - e (£>0, i=l,...,m) 
where £ is some small positive number. The constraints are 
pushed slightly into the feasible region, so the function 
will terminate in the feasible region. 

The augmented Lagrangian, L becomes 



m 



L = f + c ^ I '^i 

i=l 



(3.2) 



X. 

where ip . = max [g . + £, — — ] 

X X c 

Setting £=0, Equation 3.2 reduces to the original Lagrange 
function for the general inequality constrained problem. 
Solutions obtained with £=0 are only slightly infeasible, 
i.e., g^=+10”^; therefore, accuracy of the original Lagrange 
function was acceptable even though slightly infeasible. 
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Finally, a termination criteria was chosen such that 
the relative of absolute change in the objective function 
v;as less than 0.001 for three consecutive iterations, or 
if a preassigned number of iterations was exceeded. This 
was chosen since convergence of the objective function is 
of more interest than convergence of the multipliers, A. 
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IV. COMPUTATIONAL EXPERIENCE 



The multiplier method was used to solve five test cases. 
Three mathematical examples were used to demonstrate the 
method's computational performance as compared to other methods. 
Two engineering examples were then solved to show the applica- 
tion of the multiplier method in engineering design optimiza- 
tion. Results of the multiplier method computations are given 
in Table I. Table II shows a comparison of the multiplier 
method to other optimization methods for each case. 

A. CASE 1 : THE CONSTRAINED ROSEN-SUZUKI FUNCTION [ 5 ] 

Minimize f (x) = x^- 5 x^+X 2 - 5 x 2 + 2 x 2 - 21 X 2 +x^+ 7 x ^+50 

2 2 2 2 

such that gj^(x) = Xj^+Xj^+X2~X2+X2+X2+x^-x^-8 = 0 

g2(x).= x^-x^+2x2+X2+2x^-x^-10 ^ 0 
2 2 2 

g2(x)= 2 x^+2Xj^+X2-X2+X2~x^-5 = 0 

This problem was solved for two individual cases. Case lA 
solves the problem with constraints g^^(x) and q^(x) as equality 
constraints. Case IB treats these two constraints as in- 
equalities. The solution to each case is given in Table I. 

This problem demonstrates the method's ability to solve 
equality constrained problems directly with greater accuracy 
and efficiency. As seen in Table II, the method performs 
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no better than the exterior penalty method or CONMIN for the 
inequality constrained problem. However, when the equality 
constraints are treated directly, the multiplier method shows 
significant improvement. It should be noted that CONMIN uses 
a polynomial interpolation one-dimensional search, which is 
significantly more efficient than the Golden Section search 
in many cases. Using the polynomial search should make the 
multiplier method more efficient, if reliability can be 
preserved. The reliability of the Golden Section search made 
it the more desirable choice in the development of the method. 

B. CASE 2. A SIITLE QUADRATIC FUI^ICTICiT [23] 

2 

Minimize f(x) = 4Xj^-X2~12 

2 2 

such that gj^(x) = 25-x^-X2 = 0 

g2(x)= -10 x^+x^-10x2+X2+34 ^ 0 
(x) = -x^ 1 0 
g4(x)= -X2 ^ 0 

This case was chosen as a comparison to the Sequential 
Unconstrained Minimization Technique, SUMT [23], using an 
interior penalty function method. The problem solution is 
given in Table I. From Table II, it can be seen that the 
multiplier method is comparatively accurate to the SUMT 
method. 
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C. CASE 3: THE PAVIANI FUNCTION [23] 

2 2 2 

Minimize f (x) = 1000-x^-2x2-X2-Xj^X2-Xj^X2 

2 2 2 

such that gj^ (x) = Xj^+X2+X2-25 = 0 

g2(x)= 8 x^+4x2+7x2~56 = 0 
g^ (x) = -x^ ^ 0 

?4 = -^2 - ° 

g^ (x) = -x^ ^ 0 

In this case, it is interesting to note that even though 
the accuracy and efficiency is comparable, the penalty method 
required a larger penalty parameter, c than the multiplier 
method. This is a common point and shows the multiplier 
method's ability to avoid ill-conditioning by efficiently 
and accurately obtaining a solution with a finite c. 



D. CASE 4: THE THREE BAR TRUSS PROBLEM 

As a simple structural design problem, the three bar truss 

in Fig. 8 was considered. The problem was to determine the 

areas Al, A2 , A3 to minimize the structure weight, W. The 

design was subject to constraints -15000< a. .<20000 psi where 

13 

a. . is the stress in the truss member, i under load condition, 
13 

j . An additional geometric constraint of A1=A3 was imposed 
to maintain symmetry. 
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THE THREE BAR TRUSS PROBLEM 




MATERIAL; p = 0.1 Ib/in^ 

E = 10® psi 

GEOMETRY: H = 10 inches 

LOADS: ^1 " ^2 " 20,000 lbs. 

STRESS LIMITS: -15,000 < a,. < 20,000 psi 

’ - ij - ’ ^ 

INITIAL AREAS: A1 = A2 = A3 = 1.0 sq.in. 



FIGURE 8 . 
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E. CASE 5: CANTILEVER BEAM PROBLEM 



A cantilever beam with base, B, and height, H, variable 
over the length, L, was optimized to find the minimum volume. 

The beam, as seen in Fig. 9, was divided into five sections. 

The following constraints were imposed: (1) bending stress, 

a in each section was not to exceed ±20000 psi. (2) deflection, 
6 under the load was not to exceed ±2.0 inches, and (3) height 
to beam ratio, (H/B)<30. Additional side constraints were 
also imposed on the base and height. Separating the beam 
into five sections expanded the problem to one of ten design 
variables and 37 inequality constraints. The problem was 
solved by the multiplier method with c=1000 and The 

solution is given in Table I, with a comparison to the exterior 
penalty method and CONMIN in Table II. It should be noted 
that the exterior penalty method did not obtain a solution 
due to numerical ill-conditioning and stability problems. 

F . SUMMARY 

For the cases involving equality constraints, the advan- 
tages of the multiplier method over penalty methods and 
CONMIN has been shown. The multiplier method has proven to 
be a suitable alternative to the other methods tested for 
both equality and inequality constrained problems. While 
the method may have worked well for the cases tested, it 
cannot be guaranteed to be the most efficient, if even the 
most accurate, in all cases. The dependence of the multiplier 
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THE CANTILEVER BEAM PROBLEM 



A P=10000 lb. 



Section A-A 




The beam is broken into five segments of equal length 
and each segment is governed by two design variables for 
a total of ten design variables. 

Design requirements: 

MATERIAL; E = 30x10® psi 

GEOMETRY: Total length, L = 200 inches 

1 < H. < 30 inches i = 1,5 
— 1 — 

0.5 < B. <5 inches i = 1,5 
H. /B. < 30 i = 1,5 

i' 1 — 

STRESS LIMITS: (at the left end of each segment) 

Mr 

£ 20000 psi i = 1,5 



FIGURE 9 
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method on internal programming parameters, c and A, may 
even make the method less attractive in some cases to other 
optimization methods. 
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TABLE I 



MULTIPLIER METHOD RESULTS 

Case lA; The Equality Constrained Rosen-Suzuki Function 



Initial Obj 


= 31.000 


XI 


= (1.0, 1.0, 1.0, 1.0) 




= (-4.0, -6.0, -1.0) 


Final Obj 


= (6.0075 


XI 


= (0.01607, 1.0285, 1.9799, -1.018) 




= (-0.00515, -0.8896, 0.004387) 


Theoretical 


Optimum 


Obj 


= 6.0000 




= (0.0, 1.0, 2.0, -1.0) 


G^ 


II 

o 

0 

1 

• 

o 

o 

• 

o 



Case IB: The Inequality Constrained Rosen-Suzuki Function 



Initial Obj 


= 31.000 


XI 


o 

• 

f — \ 

o 

• 

f — \ 

o 

• 

1 — 1 

o 

• 

1—1 

II 


G^ 


o 

• 

f — 1 

o 

• 

o 

1 

o 

1 

II 


Final Obj 


= 6.1178 




= (0.03206, 1.0334, 1.9509, -1.054) 


G^ 


= (-0.1166, -0.8921, -0.1237) 



Theoretical Optimum = Same as above 
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TABLE I (contd) 



Case 2: A Simple Quadratic Function 



Initial Obj 


o 

• 

1 

II 




= (1.0, 1.0) 




= (23.0, 16.0, -1.0, -1.0) 


Final Obj 


= -31.989 




= (1.0019, 4.8986) 




= (-0.0017, -0.192, -1.026, -4.894) 



Theoretical Optimum 



Obj 


= -32.000 


X^ 


= (1.0, 4.8990) 


G^ 


= (0.0, 0.0102, -1.0, -4.899) 



Case 3: The Paviani Function 



Initial Obj 


= 976.0 


X^ 


= (210, 2.0, 2.0) 


G^ 


= (-13.0, 2.0, -2.0, -2.0, -2.0) 


Final Obj 

-T 

X 


= 961.79 

= (3.289, 0.2403, 3.7599) 


-T 

G 


= (0.0035, -0.0048, -3.286, -0.241, -3.761) 



Theoretical Optimum 



Obj 


= 961.715 


-T 

X 


= (3.512, 0.217, 3.552) 


-T 

G* 


= (0.0, 0.0, -3.512, -0.217, -3.552) 
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TABLE I (contd) 



Case 4; The Three Bar Truss 



Initial Obj 


= 


3.8284 lbs. 




= 


(1.0, 1.0, 1.0) in. 


‘^11 


= 


14142.2 psi 


^^21 


= 


8284.2 


^31 


= 


-5858.0 


0 

12 


= 


-5858.0 


0 

22 


= 


8284.2 


^32 




14142.2 


Final Obj 


= 


2.639 




= 


(0.7885, 0.4086, 0.7883) 


‘^ll 


= 


20001. 


0 

21 


= 


14636. 


0 

31 


= 


-5364.8 


a 

21 


= 


-5364.8 


'^22 


= 


14636. 


a 

32 


= 


20005. 


Theoretical 


Optirrum 


Obj 


= 


2.632 


-T 

X 


= 


(0.781, 0.424, 0.781) 



and 0^2 ^re active constraints. 
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TABLE I (contd) 



Case 5: Cantilever Beam 



Initial Obj 


= 


9000 cu 


. ft. 










= 


(15.0, 


15.0, 15 


.0, 15 


.0, 15. 


0) 


B^ 


= 


(3,0, 3 


.0, 3.0, 


3.0, 


3.0) 






- 


(17778, 


14222 , 


10667, 


7111.2 


, 3555. 


(H/B)"^ 


= 


(5.0, 5 


. 0 , 5.0, 


5.0, 


5.0) 




6 


= 


1.0535 










Final Obj 


= 


3206.2 


cu. ft. 










- 


(26.05, 


24.62, 


22.70, 


19.83, 


15.35) 


B^ 


= 


(0.886, 


0.796, 


0.733, 


0.646, 


0.506) 


Q 1 


- 


(19961, 


19904, 


19071, 


18914, 


20111) 


(H/B)"^ 


= 


(29.42, 


30.92, 


30.97, 


30.72, 


30.32) 


6 


— 


0.9528 
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COMPARISON OF VARIOUS OPTIMIZATION METHODS 
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to ill-condition 



V. CONCLUSIONS 



The multiplier method has been shown to be an accurate 
and efficient method for solving problems in engineering 
design optimization. In the particular cases tested, it 
showed comparable or improved performance to other optimiza- 
tion methods. In using a finite penalty parameter, c*^, the 
numerical ill-conditioning of penalty methods is avoided in 
most cases. The convergence of the multiplier method is at 
least linear while penalty methods converge asymtotically . 

Exact solutions are also attainable by the multiplier method. 
The ability to handle equality constraints directly makes it 
an attractive alternative to CONMIN for the equality con- 
strained problem. 

The multiplier method has other advantages which make it 
attractive. It can be used as an interior or exterior opti- 
mization method, i.e., the optimum can be approached from 
the feasible or infeasible region. Any reasonable starting 
point can be used. The dynamic selection of active constraints 
when A°= 0 is also a feature of the multiplier method. 

Research of multiplier methods is far from complete. Its 
use in the various disciplines of engineering design are 
endless. A multiplier algorithm needs to be developed which 
is independent of internal computational parameters. The 
algorithm needs to be streamlined for easy application by the 
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practicing engineer. The specific applications for which 
the multiplier method is most attractive need to be identified. 

The Augmented Lagrange Multiplier method is an extremely 
useful program in computer-aided engineering design. Its 
applications today are few, but its possibilities are endless 
and invaluable. 
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APPENDIX A 



ALGORITHM FOR GOLDEN SECTION ONE-DIMENSIONAL SEARCH 

The algorithm for the one-dimensional search subprogram 
is outlined in this appendix. The one-dimensional search is 
first performed by the Golden Section method. A cubic approx- 
imation is then performed on the last four values found by 
Golden Section method. This approach proved to be very 
effective in obtaining an accurate solution. The one- 
dimensional search is performed as follows; 

Step 1. Specify the initial interval x^ and x^, where 
X denotes the scalar, a, in Equation 3.1. 

Evaluate the functions y^ = f (x^) , y^ = f(x^). 

Step 2. Specify the absolute convergence tolerance. Ax. 

Calculate the relative convergence tolerance, 
e, and the number of function evaluations, N, 
from the equations. 

e = Ax/(x^-xj) 

N = -2.078 ln(e) + 3 

The value N is calculated in floating point 
arithmetic and rounded off to the next higher 
integer. If N<4, set N=4. 

Step 3. Calculate x^ and yi^ by the equations 

= (1 - T) Xj + TX^ 

X2 = TX^ + (1 - T) Xy 

where t = (3-/5)/2 = 0.38197 
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Evaluate = f(x^^) and ^2 ~ 

Step 4: Set counter, K=4 . This is because four functions 

have already been evaluated. If N=4 , go to 
Step 9. 



Step 5 : 
Step 6 : 



If y 2 is greater than y^ , go to Step 7. 
y^^ is greater than or equal to '^ 2 ' ^ 

lower bound. Set; 




^2 = 

'^2 - 

K = K + 1 



Go to Step 8. 

Step 7: is greater than y^^. X 2 is a new upper bound. 

Set : 




Xi = (l-T)Xj^ + TX^ 

Yl = fi^i) 

K = K + 1 

Step 8: Check convergence. If K>N, go to Step 9; else 

go to Step 5. 

Step 9: Do cubic approximation with values x^, x^^, X 2 ^ x^. 

Yc ~ ^1' ^ 2 ' ^u^ 

Step 10; Pick best of y„, Y^^/ Y 2 ' Y^* 

Y = min (y^j^, y^, y 2 / Y^,) 

X = corresponding x^, x^, X 2 or x^ 

Y is the optimum function value at location X. Stop. 
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